There are many applications for frame-to-frame tracking and/or registration of features in imagery from three-dimensional (3-D) sensors, such as LADARs (Laser Detection and Ranging), as well as from conventional two-dimensional (2-D) sensors, such as cameras, FLIRs (Forward Looking Infrared), and night vision devices. A typical LADAR emits a laser flash and then provides two types of images: a 2-D gray scale image, where each pixel represents the intensity of the laser reflection at that location; and a 3-D point cloud where each pixel represents a measure of the distance from the image plane to the object in that pixel. The term LIDAR (Light Detection and Ranging) is sometimes used for the 3-D imaging sensor. Some of the applications for frame-to-frame tracking include navigation, surveillance, geo-location, image registration and targeting.
Typical feature tracking algorithms select distinctive features in the subject images, such as edges and corners, which may be detected in subsequent frames for registration and tracking. In order to enable tracking during motion of the sensor or target, various image transformations are used to make the selected features invariant to scale. Another required transformation is feature rotation, such that the feature may be tracked from various aspect angles, as the sensor and/or target progress along a trajectory.
Once the tracked feature is successfully located in the current image frame, and matched to the same feature from previous image frames, its location in the image may be used to register the image. In a tracking or navigation scenario, the location of the tracked feature may be used as a measurement, along with measurements from other sensors performing the navigation or tracking function.
In conventional systems, the process of feature extraction and matching is a separate, computationally intensive process, which is performed externally to any extended Kalman filter (EKF) used by a navigation or tracking system. Conventional feature extraction actually strips away information that may be useful in estimating the sensor and/or target dynamics, such as image scale and aspect variation. This is especially true for 3-D imagery available from LADAR (or LIDAR) sensors. Another shortcoming of conventional systems is that the feature tracking process is vulnerable to occlusion of the features in a dynamic scenario.
As will be explained, the present invention utilizes low fidelity primitive models of some of the gross, prominent features in an image environment, as opposed to detailed, high contrast features, such as edges or corners. Tracking of these models is performed directly in the extended Kalman Filter (EKF), as opposed to needing to pre-process the image data. Any changes in scale and aspect angle with the sensor and/or target dynamics are incorporated in the model. Instead of removing the scale and aspect variation prior to inputting the image data to the EKF, the present invention utilizes the scale and aspect changes within the EKF itself, thereby enhancing the estimation of sensor and/or target location and dynamics.
As will be explained, the composite model used in the EKF is a superposition of individual primitive models that represent some dominant 3-D structures in the environment. The primitives may be easily removed or added, based on predicted start or end of occlusions. The primitive models are also mathematically well behaved functions that exist everywhere and whose derivatives exist everywhere, making them very amendable to incorporation into an EKF. These advantages, as well as other advantages of the present invention are described below.